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We propose an algorithm for creating line graphs from binary images. The algorithm 
consists of a vectorizer followed by a line detector that can handle a large variety of 
binary images and is tolerant to noise. The proposed algorithm can accurately extract 
higher-level geometry from the images lending itself well to automatic image recognition 
tasks. Our algorithm revisits the technique of image poly gonizat ion proposing a very 
robust variant based on subpixel resolution and the construction of directed paths along 
the center of the border pixels where each pixel can correspond to multiple nodes along 
one path. The algorithm has been used in the areas of chemical structure and musical 
score recognition and is available for testing at www.docnition.com. Extensive testing 
of the algorithm against commercial and noncommercial methods has been conducted 
with favorable results. 

Keywords: Line detection; image recognition; image to graph; vectorization. 



1. Introduction 

With automatic image understanding and automatic extraction of information from 
images being now one of the major challenges of web search machines and informa- 
tion extraction technology, classical tools such as vectorization and line-detection 
algorithms are being revisited as tools of choice to convert images into sets of 
graphic primitives over which pattern recognition and understanding algorithms 
can operate. 1 The ideal vectorization algorithm should produce vectors that per- 
fectly portray the information in an image, both in their position and in the number 
of high-level primitives they represent. 2 A number of factors, varying from image 
quality to the complexity of the images, affect the performance of the vectorization 
algorithms; therefore, depending on the application, different performance criteria 
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are set to improve the vectorization results. 3 A common requirement for vectoriz- 
ing binary drawings is that the vectors match the traces of the original drawing, 4 
a criterion known as maximum pixel-coincidence. 

This criterion favors the accuracy of representation vs. the extraction of high- 
level information from the image. When the vectorization is intended for image 
understanding, the identification of complete high-level structures takes priority 
over vector-pixel coincidence. 

We use vectorization to solve the problem of creating a line graph from binary 
images or drawings for automatic image understanding. The task of automatically 
understanding line drawings is becoming increasingly popular, especially since web 
search machines are now able to automatically recognize line drawings in searches. 

In our application, pixel coincidence between the images and the line graph is 
not relevant, but the extraction of complete lines and their connectivity is crit- 
ical since they represent the structural information needed for automatic image 
recognition. Using the line graph, image recognition algorithms can analyze and 
understand image content. We have successfully used our algorithm to recognize 
chemical compounds from very large databases of chemical images as well as to 
produce music from printed musical scores. 

Vectorization algorithms can be designed to work at different levels in the image: 
directly over the image pixels, 5 ' 6 over the image contours, 7 ' 8 over the image run- 
length encoding, 9 ' 10 or over the image skeleton. 11 ' 12 Although skeletonization-based 
algorithms such as the one in Ref. 13 have been, over the years, a popular way to 
vectorize text and images, the advantages and disadvantages of thinning an image 
prior to vectorization have been highly debated. 14 ' 15 The work of Sanniti di Baja 
et al 16 18 presents a valuable collection of approaches based on skeletonization. 

Skeleton-based representations are the abstraction of objects by idealized thin 
lines that retain the connectivity of the original shape, containing both shape fea- 
tures and topological structures of original objects. Skeletonization is very sensitive 
to deformations or noise on the object boundary: noise generates redundant skele- 
ton branches that disturb the topology of the skeleton graph. 19 22 Skeletonization 
is oriented to applications where the raster to vector conversion needs to maxi- 
mize pixel coincidence. As a side effect, higher- level geometric primitives (lines and 
curves) are often represented as a collection of branches. In addition, when using 
skeletonization to vectorize thick line images, the vertices that represent junctions 
between lines tend to be represented by one or more branches. 

In recent years, aiming to vectorize complex drawings containing a mixture of 
graphic primitives and text, and to allow for cluttering, authors have moved away 
from skeletonization-based techniques and have focused on pixel-based algorithms 
with stronger specific pattern recognition capabilities. Two examples of interesting 
algorithms that propose a higher-level interpretation of the images before vector- 
ization are given in Refs. 5 and 23; theses vectorization algorithms have also been 
extended to recognize arcs in drawings. 24 ' 25 In Ref. 26, an iterative algorithm is 
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proposed that starts with a coarse vectorization used to select anchor points; from 
these, the vectorization is refined based on morphological operations over the image. 

Our application requires a vectorization that ensures that the high-level struc- 
tures (lines and polygons) in the line drawings remain complete and without dis- 
tortion, so that a matching line graph can be built (matching number of edges 
and nodes). The vectorization approach must be able to work on line images with 
different thickness. In this paper, we revisit the classical technique of vectorization 
thru image polygonization followed by a novel way to extract lines from the vectors 
that allow for noise without compromising the integrity of the line structures. 

The algorithm has two main stages: first, a vector izer works over the contours of 
binary images to build vectors from contours. The vectors tend to over-poly gonize 
the binary drawings, but they are a good starting point for the second stage of line 
detection. Our algorithm is built to correctly produce a matching graph from a wide 
variety of images: varying line widths and drawing styles, and it is also robust to 
spurious components in the drawings. We have tested the algorithm on thousands 
of images obtaining over 90% of success rate. 

2. First Stage: The Vectorizer 

2.1. Overview of the vectorizer algorithm 

A block diagram of the main steps of the vectorizer algorithm is shown in Fig. 1. 
After preprocessing an image, the algorithm finds the pixels that form its contours. 
Then, the algorithm identifies the pixel edges that precisely form the image con- 
tours. From the pixel edges, the algorithm then approximates polylines that are 
finally reduced to vectors. 

For the description of the algorithm, we use an image reference system with the 
origin placed at the upper left corner. Positive x coordinates extend to the right, 
while positive y coordinates extend down. We do not use one (x, y) coordinate per 
pixel, but rather, for each pixel we work with the four coordinates of the pixel's 
vertices: (x, ?/), (x + 1, ?/), (x, y + 1), and y(x + 1, y + 1). 

The algorithm works on binarized images lb (x,y). The images are first seg- 
mented into connected components, which are groups of foreground pixels that are 
connected at least by a vertex. The vectorization takes place on each connected 
component independently. For clarity, Table 1 contains the abbreviations that will 
be used while describing the vectorization algorithm. 
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Fig. 1. Block diagram of the vectorizer. 
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Table 1. Abbreviations used in the description of the vectorizer. 



Abbreviation 


Meaning 


CE 


Contour edge 


CEfn 


Front neighbor of a contour edge 


PC 


Pixel center 


V, VI, V2 


Vectors 


VI', V2' 


Segments of vectors VI, V2 ... 


S 


Succession of pixel centers 


VA, VB, VC 


Vectors that form edges of a quadrilateral 


Vset 


Set of vectors that can form the edge of a quadrilateral 


L, LI, L2 


Lines 



2.2. Contour pixels and contour edges 

Contour pixels are foreground pixels having at least one first-degree neighbor pixel 
(neighbor pixel sharing an edge) that is a background pixel. Figure 2 shows a con- 
nected component of Ib(x, y) and the contour pixels of the connected component. 

A contour edge, CE, is the edge of a contour pixel that is also shared by a 
first-degree neighbor pixel that belongs to the background. A contour pixel can 
contain 1, 2, 3, or 4 contour edges. We classify contour edges in four types shown in 
Fig. 3: 

• Type 1: From (x,y) to (x + 1,2/); shared by a contour pixel and a background 
pixel above it. 

• Type 2: From (x,y + 1) to (x + l,y + 1); shared by a contour pixel and a 
background pixel below it. 

• Type 3: From (x,y) to (x,y + 1); shared by a contour pixel and a background 
pixel to its left. 

• Type 4: From (x + l,y) to (x + l,y + 1); shared by a contour pixel and a 
background pixel to its right. 
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Fig. 2. (a) A connected component of Ib(x, y) and (b) the contour pixels of the connected 
component. 



Vectorization and Line Detection for Automatic Image Recognition 443 









T/jM 3 <— 








Ty|»2 





Fig. 3. The four types of contour edges. Arrows indicate the direction of the first-degree neighbor 
background pixel. 

2.3. Contour- edge polylines 

The contour edges in the connected components are joined in polylines. The 
sequence of contour edges are sequentially numbered to create a path along the 
polylines as shown in Fig. 4. 

A contour edge has at least two neighboring contour edges, one at each vertex, 
but it can have up to three neighboring contour edges per vertex as shown in Fig. 5. 
In order to create a single path along the polyline, the algorithm must choose only 
one neighboring contour edge when more than one is encountered at a vertex; in 
such cases, the following rules, also illustrated in Fig. 5, are applied. 

Let CE be a contour edge that goes from PI with coordinates (xl,yl) to P2 
with coordinates (x2,y2) where xl < x2 and yl < y2. 

If CE is joined to more than one neighboring contour edge at P2: 

(1) If CE is horizontal, a vertical contour edge shared by a background pixel to its 
right is chosen. 

(2) If CE is vertical, a horizontal contour edge shared by a background pixel below 
it is chosen. 

If CE is joined to more than one neighboring contour edge at PI: 

(1) If CE is horizontal, a vertical contour edge shared by a background pixel to its 
left is chosen. 

(2) If CE is vertical, a horizontal contour edge shared by a background pixel above 
it is chosen. 

The contour edge polylines can be seen as ordered sequential chaincodes that 
are built according to the rules illustrated in Fig. 5. Their main difference with 
classical chain codes is that the edges in the polylines are not labeled according 
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Fig. 4. Contour polylines created from contour edges. A contour pixel can have 1, 2, 3, or 4 
contour edges, (a) Original image with one connected component; (b) contour pixels in dark gray; 
(c) polyline created from contour edges; (d) for each contour pixel, we keep track of the number of 
contour edges belonging to it; and (e) path along the polyline created by sequentially numbering 
the contour edges. 




(a) (b) (c) 

Fig. 5. Example of tracing a path along a polyline, (a) Original image, (b) CE 1 has three possible 
neighboring contour edges at one vertex identified by arrows a, b, and c. Only one contour edge 
can be chosen as neighbor in order to establish a directed path, (c) The final path along the 
polyline. 
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to a meaningful code (that represents their direction, for example), but rather, are 
labeled sequentially to represent a guided path along them. 

2.4. Front neighbor of a contour edge 

As the algorithm can, later on, establish the correct neighborhood information 
between all the polylines that are generated in an image, we define the concept of 
front neighbor of a contour edge, CEfn, for any give CE, as follows: 

If a CE is of type 1 (shared by a background pixel above it), going from (x, y) 
to {x + l,y), its CEfn is defined as the first contour edge in the image that lies 
below CE, going from (x, y + k) to (x + 1, y + fc), with k an integer > 1, and that 
is of Type 2 (shared by a background pixel below it). 

If a CE is of type 2 (shared by a background pixel below it), going from (x, y) 
to (x + 1,2/), its CEfn is the first contour edge in the image that lies above CE, 
going from (x,y — k) to {x + l,y — fc), where k is an integer > 1, and that is of 
Type 1 (shared by a background pixel above it). 

If a CE is of type 3 (shared by a background pixel to its left), going from (x, y) 
to (x,y + 1), its CEfn is the first contour edge in the image that lies to the right 
of CE, going from (x + k, y) to (x + k,y+l), with k an integer > 1, and that is of 
Type 4 (shared by a background pixel to its right). 

If a CE is of type 4 (shared by a background pixel to its right), going from 
(x, y) to (x, y + 1), its CEfn is the first contour edge in the image that lies to the 
left of CE, going from (x — k,y) to (x — k, y + 1), with k an integer > 1, and that 
is of Type 3 (shared by a background pixel to its left). 

If CE1 has as its CEfn the contour edge CE2, then CE2 has as its CEfn the 
contour edge CE1. The CEfns of the contour edges in a connected component are 
illustrated in Fig. 6. 

If at least one contour edge CE1 in polyline A has as its CEfn a contour 
edge CE2 that belongs to polyline B, then polylines A and B were formed from a 
single connected component in the original image. It is said that polylines A and B 
are neighbors. If polyline B is also neighbor of polyline C, then polylines A and C 
are also neighbors. A connected component contains a set of neighboring polylines. 
The neighborhoods among polylines are illustrated in Fig. 7. 

2.5. Pixel center polylines 

The contour-edge polylines and the directed paths along them will be used as 
auxiliary information to build vectors that better reflect the higher-level geometrical 
information of the image. By traversing the path along a contour polyline a weighted 
pixel center, PC, will be defined for every contour pixel that generated the contour 
polylines in the first place. A PC is located at the center coordinates of a pixel as 
shown in Fig. 8. 

Weighted PCs are assigned to contour pixels as follows: While traversing the 
directed path along the contour edge polylines, for every contour edge in the polyline 
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(a) (b) 

Fig. 6. Arrows indicate the front neighbors of the contour edges, CEfns. (a) CEfns of contour 
edges types 1 and 2 and (b) CEfns of contour edges of types 3 and 4. 
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Fig. 7. Establishing the neighborhood among polylines of contour edges, (a) Original image with 
one connected component and two neighboring polylines; (b) the two polylines; (c) the outermost 
polyline A; (d) the innermost polyline B; and (e) polyline A consists of 12 contour edges (Ax) and 
polyline B consists of 4 contour edges (Bx). Contour edges A2, A5, AS, and All have as CEfns the 
contour edges Bl, £>2, £>3, and B4, respectively, therefore, establishing the neighborhood between 
polylines A and B. 
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Fig. 8. Pixel center of a pixel, P. The algorithm places a PC in the center coordinates of contour 
pixels. If a pixel has upper left coordinates (x, y), then the PC has coordinates (x + 0.5, y + 0.5). 

a PC will be assigned to the contour pixel that generated it. If a sequence of 
contour edges in the polyline were generated by the same contour pixel, only one 
PC is assigned to the contour pixel. However, if two contour edges were generated 
from the same contour pixel, but are not sequential in the directed path along the 
polyline, each contour edge contributes one PC to the contour pixel, originating a 
double PC. Similarly, if three or four contour edges in the polyline originated from 
one contour pixel, but are not sequential in the directed path along the polyline, 
each contour edge contributes one PC to the contour pixel generating a triple or 
quadruple PC in a contour pixel. Figure 9 shows an example of the generation of 
the PCs from the polyline of contour edges. The PCs include a double PC created 
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Fig. 9. Generation of PCs from contour edges. PCs are shown with a plus symbol inside every 
contour pixel. A number (N) besides the plus symbol "+(iV)" indicates there are iV PCs in that 
contour pixel. 
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from two contour edges (PCs 3 and 8) that were generated from the same contour 
pixel but that are not sequential in the directed path that traverses the polyline. 

Once the weighted PCs have been assigned, a new polyline with a directed path 
is established that traverses the PCs. A single PC is only visited once along the 
directed path, while a double PC is visited twice along the directed path. A triple 
PC is visited three times and a quadruple PC four times. The order in which the 
PCs along the directed path are visited is given by the contour edges that created 
the PCs. That is, the order of the directed path along the PCs is given by the order 
of the directed path that was first created along the contour edges. The directed 
path along the PCs shown in Fig. 9 is shown in Fig. 10. Figure 11 shows an example 
of a polyline of PCs that contains a quadruple PC, this PC corresponds to four 
vertices of the polyline. The four vertices are non-sequential along the directed path 
that traverses the polyline. 

2.6. Vector polylines 

From the polylines of PCs, the algorithm finally creates polylines of vectors. The 
vectors reflect the higher-level geometry in the original image and can be used to 
extract geometrical primitives such as lines, or to construct Bezier curves for CAD 
and digital art applications. Although the main objective of the algorithm is to 
detect lines, some successful tests with curves have been carried out. Examples of 
our tests are shown later in the results section. 

Vectors go from one PC to another. Let V be a vector that goes from PCI 
(#1,2/1), to PC5 (#2,2/2), and let S be the succession of PCs between PCI and 
PC5. The vector V is a valid vector if all the PCs in S are at a distance d(PCx,V) 
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Fig. 10. Ordered polyline along the PCs. The polyline contains a double PC (PCs 3 and 5) that 
is visited twice along the directed path, that is, the double PC represents two vertices of the 
polyline. 
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Fig. 11. (a) A connected component with all contour pixels and the ordered contour edges. The 
polyline of contour edges is used to generate a new polyline of PCs. Contour edges 3, 8, 13, and 18 
are not sequential along the directed path that traverses the contour edges and, therefore, generate 
a quadruple PC in the contour pixel that generated them, (b) The ordered polyline along the PCs. 
The quadruple PC corresponds to vertices 2, 3, 6, and 8 of the polyline. 



from V less than or equal to a parameter Dmax. The distance between a PCx 
(xO,yO) in S and vector V is calculated by Eq. (1): 



d(P,V) 



\{x2 - xi){yi - y ) - (zi - x )(y 2 - yi)\ 
V fa - xxf + (y 2 - yif 



(1) 



Figure 12 illustrates the formation of a vector V going from PCI to PC5 and 
the set S of intermediate PCs at a distance d = Dmax from V. 
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Fig. 12. Formation of a vector. Given a sequence of PCs, a vector can be formed from the first to 
the last PCs in the sequence if the distance of all intermediate PCs in the sequence to the vector 
is <= to a parameter Dmax. (a) The sequence of five PCs and (b) formation of vector V. 
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Given Dmax, each polyline of PCs will be transformed into a polyline of vectors 
by visiting the PCs orderly along the directed path. A change in the order in which 
the PCs are vectorized, changes the vectorization results. However, the changes 
are minimal in that they do not affect the line detection results. The parameter 
Dmax controls how sensitive the vectors are to changes in direction in the contour 
pixels. A larger Dmax reduces sensitivity to direction changes and produces fewer 
vectors. Figure 13 shows an example of a polyline of PCs that is transformed into 
a polyline of vectors. Figure 14 shows an example of a polyline of vectors formed 
over character "a." Figure 15 shows two examples of the same image at different 
resolutions, and the resulting polylines of vectors created using varying values of 
Dmax. 



2.7. Vector postprocessing 

An optional step, depending on the application, is vector postprocessing. When 
an image has to be described by a matching graph, and every line in the image 
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Fig. 13. Vectorization algorithm. A polyline of vectors is created from a polyline of PCs. In (a), 
only the PCs are shown and in (b), the vector polyline is shown. 




(a) (b) 

Fig. 14. (a) Two polylines of PCs (only the PCs are shown, not the connecting edges) and (b) two 
polylines of vectors formed from the two polylines in (a) using a Dmax = 0.4. 
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(a) (b) (c) (d) 




(e) (f) (g) (h) 

Fig. 15. (a) and (e) The same image at two different resolutions. In (a), the image is 181 X 270 
pixels, while in (e), the image is 27 x 44 pixels; (b) and (f) polylines of vectors with Dmax = 
0.5; (c) and (g) polylines of vectors with Dmax = 1.0 and (d) and (h) polylines of vectors with 
Dmax = 2.7. 

must be described as a single entity, rather than a set of connected small lines, 
it has proven useful to postprocess the vectors using two additional parameters: 
Qunion and Dunion. Qunion controls the maximum degree of parallelism between 
two neighbor vectors to still be considered different vectors. The angle between two 
neighbor vectors VI and V2 joined by vertex P is calculated as in Eq. (2): 

co S <V-,, K )=co S --(p^_), (2) 

where Vl x is vector VI with its second vertex (not P) translated to the origin and 
V2' is V2 with vertex P translated to the origin. 

When (VI, V2) = qunion, the two vectors are joined in a new vector Vunion. 
The creation of a new Vunion, however, cannot be directly accepted without val- 
idating that Vunion remains at an acceptable distance Dunion > Dmax from 
all the PCs that generated VI and V2. If Vunion is accepted, the algorithm can 
continue to join Vunion with its neighbor vectors in successive iterations as long 
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Fig. 16. Example of the joining of two vectors into Vunion and the parameters qunion and 
Dunion. 

as ^(Vunion, Vx) < qunion and the distance from Vunion to all the PCs that 
generated VI and V2 < Dunion. Figure 16 shows an example of the joining of 
two vectors into Vunion and the parameters qunion and Dunion. 

3. Line Extraction Algorithm 

3.1. Line extraction 

Using the vector polylines, the algorithm extracts higher-level geometry primitives 
from the image. We are interested in describing binary images in terms of line 
graphs; therefore, we will extract lines from the vectors. To extract lines from 
the polylines of vectors, the algorithm creates quadrilaterals with two opposing 
edges formed from vectors or portions of vectors in the polyline, we call these 
two opposing edges in any quadrilateral VA and VB. The other two edges in the 
quadrilaterals are imaginary vectors (they do not belong to the vector polyline) that 
are perpendicular to the edge VA and join VA to VB, we call these imaginary edges 
VC and VD. From a quadrilateral, the algorithm extracts the line that extends 
from the middle point of VC to the middle point of VD. Figure 17 illustrates the 
procedure of extracting lines from quadrilaterals of vectors. In order to carry out 
the line detection from quadrilaterals, the algorithm tests every vector in a vector 
polyline to find out whether it can be part of one or more quadrilaterals; the details 
of how quadrilaterals are formed are given in the next section. 

3.2. Finding vectors to form quadrilaterals 

To form quadrilaterals, the algorithm tests every vector in the vector polylines. For 
each vector, the algorithm first verifies that it has not already been completely used 
to form a quadrilateral. If a vector or a segment of it has not yet participated in 
the creation of a quadrilateral, the vector or segment of it will be considered as a 
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(d) (e) (f) (g) 



Fig. 17. Line detection from quadrilaterals, (a) Original image; (b) polyline of vectors; (c) lines 
detected in the image and (d) to (g) details of line detection: each line is extracted from a quadri- 
lateral with two edges VA and VB coming from the vector polyline and two edges VC and VD 
that connect VA and VB. For each line, the algorithm can also indicate its average pixel thickness 
in the original image and its slope. 

candidate VA edge to form one. To find the opposing VB edge for the candidate VA, 
the algorithm finds a set of candidate VB vectors VBset. One way to find VBset is 
to iterate through all the vector polylines in one connected component, but a much 
faster way is to use the information of the CEfns. 

The vector polylines are formed from PC polylines, which are in turn formed 
from CE polylines. Thus, there is a relation between vectors — > PCs — > CEs. 
VBset contains the vectors that were formed from the set of CEfns of the CEs that 
originated VA. The flow of information to find VBset starting from VA is shown in 
Fig. 18. 

Figure 19(a) shows a polyline of vectors Fig. 19(b) shows a vector VA along with 
the CEs that originated it. Figure 19(c) shows the CEfns for VA and Fig. 19(d) 
shows the set of candidate front vectors VBset. 

The opposing edge VB that will form a quadrilateral with VA is formed from 
one or more vectors in VBset. The vector or vectors to become VB from VBset are 
those that have not participated in the formation of any other quadrilateral, and 
satisfy the following two conditions: 

Condition 1: Suitable VB vectors are vectors in VBset that lie at least partially 
inside a square called CA having VA as an edge and extending in the direction 
of the foreground pixels in the image. Figure 19(e) shows square CA created from 
vector VA. The vectors in VBset that satisfy this condition form VBsubset. 
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Fig. 18. Starting from a vector VA, the algorithm goes through the CEs that originated VA, 
then through the set of front neighbor contour edges, CEfns, then through the set of vectors that 
were formed from the CEfns and finally finds a suitable vector to become VB from that set. 




(a) (b) (c) 




(d) (e) (f) (g) 

Fig. 19. Conditions for forming a quadrilateral starting from VA. (a) A polyline of vectors, (b) A 
vector VA and the CEs that originated it. (c) The CEfns of VA. (d) The set of vectors VB that 
were created from the CEfns: B, C, D, E, and F. (e) The first condition to become a front vector 
VB of VA is that the vectors be inside a square CA that has VA as an edge and extends toward 
the foreground pixels, vectors B, C, and D fulfill this condition, (f) Vectors B, C, and D fulfill the 
first condition, (g) Vectors that fulfill the second condition of being closest to VA and not being 
perpendicular to VA are vectors C and D. 



Vectorization and Line Detection for Automatic Image Recognition 455 



Condition 2: From VBsubset, the algorithm chooses the vector or vectors that 
are closest to VA. To determine which vectors are closest to VA, the vectors in 
VBsubset go through the following steps: 

(1) VA is divided in N points with a separation q = 1/3 pixels between points. TV 
can, therefore, be calculated as TV = floor (||VA||/g) + 1. 

(2) The vectors in VBsubset are projected over VA. Vectors in VBsubset that are 
perpendicular to VA have no projection and are eliminated from VBsubset . 

(3) For each point n on VA, we calculate the point- vector distance to all the vectors 
in VBsubset whose projection over VA includes point n. 

(4) The segments of vectors in VBsubset that were closest to at least one point 
n on VA are the final candidate VB vectors and they can participate in the 
formation of one or more quadrilaterals with VA, as illustrated in Fig. 20. 
In Fig. 20(a) VBsubset contains two vectors VI and V2. Figure 20(b) shows 
that the complete vector VI is closest to all the points in the segment VA'l 
of VA. Fig. 20(c) shows that only V2', a segment of V2, is closest to the 
points in the segment VA'2 of VA. For this example, vector VI and seg- 
ment V2 ; will participate in the formation of two quadrilaterals with VA'l and 
VA'2. 




Fig. 20. Example where the set of VB vectors includes one full vector and a segment of a vector. 
In this example, the whole vector VI is closest to points over VA, while only a segment V2 ; of 
vector V2 is closest to points over VA. 
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3.3. Forming quadrilaterals 

A vector VA and the set of candidate VB vectors can form one or more quadrilat- 
erals. In general, a vector VA and a set of N candidate VB vectors can create M 
quadrilaterals, where M < N . From the M quadrilaterals K lines can be formed, 
where K < M. Figure 21 shows an example of a VA that formed two quadrilaterals 
and two lines. Quadrilaterals are formed as follows: 

From the set of candidate VB vectors, the algorithm chooses those that satisfy 
Eq. (3), 

| cos(6 V Bx - V a)\ > P , (3) 

where #VBx is the angle of one candidate VB vector with respect to the origin, 
6YA is the angle of VA with respect to the origin, and Op is a parameter angle that 
controls how parallel vectors VA and VBx are (in our case Op = 0.7). 

• From the vectors chosen in the previous step, the algorithm joins those that are 
neighbors. To join two neighbor VB vectors, the common vertex is discarded 
and a new VB vector is formed using the two extreme vertices of the original VB 
vectors. At this point, the candidate VB vectors are ready to form quadrilaterals. 

• Each candidate VB vector will form one quadrilateral with VA. The quadrilat- 
eral's edges are defined as follows: 

• First edge: The candidate VB vector. 

• Second edge: The projection of the candidate VB vector over VA. 

• Third and fourth edges: Two vectors perpendicular to the second edge, that 
extend toward the first edge. We call these edges VC and VD. 




Fig. 21. The process of line formation from quadrilaterals from a vector VA. (a) Two polylines 
of vectors in one connected component, (b) Vector VA and its set of VB vectors formed by vectors 
VI, V2', V3', and V4. V2' and V3' are segments of vectors V2 and V3. (c) V2' is not parallel 
enough to VA and is discarded from the set of VBs, thus, only three quadrilaterals are formed: 
CI, C2, and C3. (d) C3 is not a valid quadrilateral because its C and D edges are longer than the 
VA-VB edges (VA'3 and V3'). (e) Two lines are formed from CI and C2. (f) The final lines. 
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The first and second edges have similar angles. The third and fourth edges are 
parallel between them. In the case that VA and VB are neighbors, the third and/or 
fourth edge can have zero length. 

Finally, the quadrilaterals are validated. To pass validation, edges VC and VD 
must be shorter than the projection of VB over VA. Valid quadrilaterals then form 
lines that extend from the midpoint of VC to the midpoint of VD. Figure 21 illus- 
trates the whole process of forming lines starting with a VA vector. 

Two sides of the quadrilateral can cross each other and one or two sides can 
have zero length. This happens when thin lines (generally 1 pixel thick) are present 
in the image. The methodology to detect lines is the same whether the sides of the 
quadrilateral cross each other or not or any of the sides have zero length. Figure 22 
illustrates a case when a 1-pixel-thick line is detected. 

Vectors or segments of vectors that participate in the formation of a line can 
no longer participate in the formation of other quadrilaterals or lines. That means 
that a vector or a segment of a vector can create one line at most. If a segment of 
a vector has already been used to create a line, the remaining portion of the vector 
can still be used to create one more line or, if it is divided into segments, each 
segment can contribute to one line. In Fig. 21, the remaining parts of VA that did 
not contribute to the quadrilaterals, can still be used to form lines. Figure 23 shows 
another example of line formation where two of the candidate VB vectors are neigh- 
bors and, therefore, joined before forming a quadrilateral. At the end of the process, 
all the vectors in the polylines will have been tested for line formation, although 




(d) (e) 

Fig. 22. Detection of thin lines (1 pixel thick), (a) Original image; (b) vectorization result; 
(c) formation of quadrilaterals: one quadrilateral has two sides that cross each other and two 
quadrilaterals have one and two zero length sides; (d) line detection and (e) line graph construction. 
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Fig. 23. An example of line formation (a) Polyline of vectors; (b) Vector VA and its set of 
candidate VB vectors: VI, V2, and V3 7 . V3 7 is a segment of vector V3. (c) Vector VI is not 
parallel enough to VA and is discarded. Vectors V2 and V3 ; have a similar angle to VA and are 
neighbors, so they are joined in only one vector Vjoint. (d) A quadrilateral is formed from Vjoint, 
its projection over VA and two parallel vectors that extend from the vertices of Vjoint to its 
projection over VA. 

it can be the case that a small remaining portion of them did not contribute to 
any line. 

3.4. Line graph 

For some applications, where higher- level information is to be extracted from the 
images, it is necessary to build a connected graph from the detected lines. 

In order to form the graph, the algorithm has to assign the proper line connec- 
tivity by finding neighbors between the lines as follows: Be L a line that was created 
from quadrilateral C, where L extends from point PI to point P2. C has edges A, 
B, C and D, and vertices vl, v2, v3, and v4 as illustrated in Figs. 23(b) and 23(c). 
The neighbor lines of L are found as follows: 

On PI: 

(1) Starting at vertex vl, and in the direction away from quadrilateral C, the algo- 
rithm sequentially visits the vectors along the vector polyline until it finds one 
vector that has participated in the formation of a quadrilateral and a line Ln. 
Ln is then a neighbor to L. In Figs. 24(c) and 24(d), starting the neighbor- 
hood search at vertex vl, vector V6 is the first vector encountered that has 
participated in the formation of a line, L4. L4 is a neighbor of L. 

(2) The steps in (1) are repeated starting at vertex v2. In Figs. 24(c) and 24(d), 
starting the neighborhood search at vertex v2, vector V4 is the first vector 
encountered that has participated in the formation of a line, L6. L6 is a neighbor 
of L. 
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(d) (e) (f) 

Fig. 24. Example of how the neighborhoods between lines are found, (a) The process starts from 
the lines created and the vectors that contributed to each line, (b) Quadrilateral that originated 
L. The edges of the quadrilateral are shown as A, B, C, and D and the vertices as vl, v2, v3, and 
v4. (c) Starting at each vertex in the quadrilateral, the algorithm finds a vector along the vector 
polyline that has participated in the creation of a line Ln. For this example, such vectors are V2, 
V4, V6, and V8. (d) The lines originated from V2, V4, V6, and V8 will be neighbor lines to L. 
In this example, the neighbor lines are shown as LI, L3, L4, and L5. (e) Creation of two new 
vertices Ul and U2 where the neighbor lines will be joined to each other. Ul has as coordinates 
the average of the vertices u2, u3, and u4 shown in the figure and in a similar manner, U2 has the 
average coordinates of the individual vertices ul, u5, and u6. (f) Neighbor lines joined at the new 
vertices. 

On P2: 

(1) Starting at vertex v3, and in the direction away from quadrilateral C, the algo- 
rithm sequentially visits the vectors along the vector polyline until it finds one 
vector that has participated in the formation of a quadrilateral and a line Ln. 
Ln is then a neighbor to L. In Figs. 24(c) and 24(d), starting the neighbor- 
hood search at vertex v3, vector V8 is the first vector encountered that has 
participated in the formation of a line, L5. L5 is a neighbor of L. 

(2) The steps in (1) are repeated starting at vertex v4. In Figs. 24(c) and 24(d), 
starting the neighborhood search at vertex v4, vector V2 is the first vector 
encountered that participated in the formation of a line, LI. LI is a neighbor 
of L. 

For terminal lines, the algorithm will find no neighboring lines. 

Once the basic neighborhood between the lines has been established, a second 
pass over the lines is made to complete the neighborhoods as follows: if LO is a 
neighbor of LI at a vertex vn, and LI is a neighbor of L2 at the same vertex vn, 
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(a) (b) (c) 

Fig. 25. Six lines joined at a new vertex U. (a) The six lines created from quadrilaterals, (b) New 
vertex U created by averaging vertices ul thru u6. (c) The line graph. 




(a) (b) 

Fig. 26. Merging of lines with similar angles in the line graph, (a) Lines detected from quadri- 
laterals, (b) Line graph including the merging of lines with similar angles. 

then L0 is a neighbor of L2 and L2 is a neighbor of L0. Once the neighborhoods are 
established, all the lines that are neighbors at a vertex vn can be physically joined 
at a new vertex as is illustrated in Fig. 25. 

3.5. Line postprocessing 

Once a line graph is created, it can be useful to merge neighbor lines having similar 
angles as was done in the vector postprocessing stage. This has proved very helpful 
when working with noisy images. Figure 26 shows an example of a noisy image and 
the merging of neighbor lines having similar angles. 

4. Results and Performance 

The accuracy with which the algorithm detects the thickness, angle, and length 
of each extracted line was measured. Test images with isolated lines were used for 
each case. The results show that the algorithm detects the line's properties with 
an accuracy of 99.97% for the angle, 95.95% for the length, and 85.66% for the 
thickness. The test images are shown in Fig. 27. The mean of the processing time 
of the line detection algorithm is between 83% and 124% of the processing time of 
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(h) 

Fig. 27. Test images and results. The images were used to measure the accuracy with which the 
proposed algorithm detects line's properties: (a) angle; (c) and (d) length; (g) thickness and (b), 
(e), (f), and (h) results. 



the vectorization algorithm. The mean of the processing time of the vectorization 
algorithm is between 20% and 60% of the processing time of Adobe Illustrator. The 
results of the line detection algorithm were compared against the standard Hough 
transform (STH). Three advantages of the proposed algorithm over the STH are 
that the former does not require to predefine the rho and theta domains and the 
number of lines (the number of peaks in the STH matrix) to be detected; it does 
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not require to skeletonize the image, so that the image is not modified; and that 
it detects the line's thickness and connection with other lines. A comparison of the 
results of our algorithm vs. the SHT is shown in Figs. 28 and 29. In these figures, it 
can be seen that the STH introduces noise (inexistent lines) when trying to detect 
all the lines in an image. The noise is present because, in general, it is necessary 
to choose a bigger number of peaks in the STH matrix than the number of lines in 
the image. 

The proposed algorithm is able to detect lines in noisy images with no or min- 
imum modification of parameters. Examples are shown in Fig. 30. The method is 
flexible to adapt to a specific type of noisy images with the modification of two 
parameters: Dmax and 6 join. 

Figure 31 illustrates the results of each step of the line detection algorithm. 




(a) 

Fig. 28. Line detection: (a) original image and (b) and (c) detected lines with the proposed 
algorithm and the SHT, respectively. Arrows in (a) indicate the lines that could not be detected 
with the SHT when selecting 60 peaks of the Hough matrix. 
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(c) (f) 

Fig. 30. Line detection in a noisy image: (a) and (d) original image; (b) and (e) detected lines; 
and (c) and (f) joined lines. 



Current applications are chemical structure and music score recognition. Results 
are shown in Figs. 32 and 33. 

Figure 34 shows a comparison of the results of a skeletonization algorithm with 
the vectorization algorithm used in the paper. Skeletonization was performed using 
a MATLAB built-in algorithm and no postprocessing was done on the results. 
Because skeletonization emphasizes pixel coincidence of the vectors with the image, 
it requires postprocessing of the skeletons if it is to be used for building a graph 
from a line image. For example, in skeletonization, vertices in thick images are 
represented as branches. In addition, skeletonization is influenced by the thickness 



Vectorization and Line Detection for Automatic Image Recognition 465 




(e) 

Fig. 31. Results of each step of the line detection algorithm: (a) original image; (b) vectorization 
result; (c) formation of quadrilaterals; (d) line detection from the quadrilaterals; and (e) line graph 
construction. 



of the line images; thicker line images generate additional branches that must be 
pruned. 

For self-testing, our vectorization and line detection algorithms are available 
in an easy-to-use program that can be downloaded from the project's Web site 
www.docnition.com. 
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(c) (f) 

Fig. 32. Graph that represents a chemical structure: (a)-(c) original images and (d)-(f) graph of 
chemical bonds and atoms that were created using the line detection algorithm as the first stage 
in the pipeline. 
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Fig. 33. Segmentation of a music score: (a) and (d) original image; (b) and (e) detected stave 
represented with detected and filtered lines in the image; and (c) and (f) detected music symbols 
that are delimited through vectors. 
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Fig. 34. Comparison between skeletonization (without postprocessing) and vectorization of line 
images. The first column shows three examples of thick-line images. The second column shows the 
result of applying a MATLAB skeletonization to the images. The presence of spurious branches 
that need to be pruned can be seen. In addition, vertices are represented by one or more vectors. 
The third column shows the result of vectorizing the images. The fourth column shows the line 
graph built from the vectorization. 



5. Conclusion 

In this paper, we propose a methodology for creating line graphs from binary 
images. The methodology capitalizes on the contour information of the binary 
images to create a vectorization that remains close to the original traces. From 
the vectorization, the algorithm extracts lines by creating quadrilaterals from vec- 
tor segments; this approach has proven very robust to preserve the higher-level 
geometry in binary images, even in the presence of noise. The resulting line graphs 
lend themselves to image understanding applications and have been tested in the 
areas of chemical molecule recognition and music score recognition. The complete 
system is available on-line for free testing. 
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